HI @LEDfan ,
i've have a first running version of shinyproxy, containerproxy working with Snowflake Snowpark Container Services
https://docs.snowflake.com/en/developer-guide/snowpark-container-services/overview

have a couple questions

1. i need to solve how to dynamically update http header token coming into shinyproxy onto outgoing http headers on users containers. this is being done at container create time, but not sure how to refresh it.

2. i noticed the AWS ECS backend stores labels as ECS task env variables. is it consider safe to store all labels as env variables? also i noticed AWS ECS backend doesn't implement recovery.

An example configuration is

proxy:
  title: Open Analytics Shiny Proxy
  logo-url: https://www.openanalytics.eu/shinyproxy/logo.png
  landing-page: /
  heartbeat-rate: 10000
  heartbeat-timeout: 300000
  port: 8080
  authentication: spcs
  container-backend: spcs
  stop-proxies-on-shutdown: false
  recover-running-proxies: true
  admin-groups: <snowflake role(s) admin users are granted to>
  spcs:
    compute-pool: "my_compute_pool"

  specs:
    - id: 01_hello_sf
      container-image: "/MYDB/images/shinyproxy/shinyproxy-integration-test-app:latest"
      container-cmd: [ "R", "-e", "shinyproxy::run_01_hello()" ]
      container-memory-request: 2048
      container-cpu-request: 1024
      port: 3838

Another question, the Snowflake Java client was generated from the snowflake openapi specification github project using okhttp3.
https://github.com/snowflakedb/snowflake-rest-api-specs

is there a preferred java http library to use to keep the package sizes and dependencies as small as possible?

Hi

Thanks a lot for the effort! We are always happy to add new backends, the only problem I see is that we don't have any snowflake deployments ourself. In order to guarantee maintenance on this backend, we need to look into whether we can use some trial or development license.

i need to solve how to dynamically update http header token coming into shinyproxy onto outgoing http headers on users containers. this is being done at container create time, but not sure how to refresh it.

In the current implementation,the headers that are send to the application are not updated after the application has been started. This can be changed, although I'm not yet sure about the exact implementation. The headers are injected here: https://github.com/openanalytics/containerproxy/blob/master/src/main/java/eu/openanalytics/containerproxy/util/ProxyMappingManager.java#L260-L261

i noticed the AWS ECS backend stores labels as ECS task env variables. is it consider safe to store all labels as env variables? also i noticed AWS ECS backend doesn't implement recovery.

I believe these are added as labels in the task. But in general we consider it safe to add them to labels in the backend, but I wold not add them as environment variables (since the app should not see all these values).

is there a preferred java http library to use to keep the package sizes and dependencies as small as possible?

okhttp3 is already a dependency, so it's fine to use. However, we use jackson for json processing, so if possible it would be better to use jackson instead of gson.

I don't have lots of experience with Snowflake, could you elaborate your use-case of running ShinyProxy on Snowflake?
What are the advantages? I'm also wondering how the initial deployment of ShinyProxy is done. Is ShinyProxy running on the Snowflake cluster, or is it deploy externally?

typical deployment is shinyproxy running as snowflake service, then creates additional private services for users apps.

the code to support shinyproxy external is to make development easier and faster. i dont think it will ever be used this way in production.

use cases and benefits

1. makes the user auth really easy. inherits the snowflake identity setup.
2. can use pass through of users identity or service role to access the data
3. analytics with python or R. snowflake has an existing Notebook feature. python or SQL. some users like it, some don't.
   eg codeserver. how does one manage creating a codeserver snowflake service per user...
4. data apps with python or R. snowflake owns and supports streamlit in snowflake. is not meant to solve everything. snowflake has marketplace for vendor apps. half way is supporting other python/R data app frameworks.

the list of openapi java codegen which have Jackson json are here with different webclients. any of these won't introduce additional packages?

search for "library template (sub-template) to use"
https://openapi-generator.tech/docs/generators/java

will add an example SQL to setup the environment soon.

It seems to me that the webclient is the best fit, this uses the (newest) Spring API to make requests.

also i noticed AWS ECS backend doesn't implement recovery.

I forgot to reply to this question: we usually don't implement app recovery for new backends. The operator is now available for both Docker and Kubernetes, and provides a much better experience. We might consider removing the app recovery feature.

thanks will try redo the openapi java client soon.
and relook at the SPCS user/client token pass through on http header.

SPCS doesn't expose kubernetes. SPCS is closer to aws ECS, so the shinyproxy operator is not useable.
though i can tell inside a snowflake service container, its actually a k8 node; they abstract that away.
but at least SPSC is consistent across AWS, Azure and GCP.

i've found my backend restore implementation on SPCS makes it very easy to deploy shinyproxy without needing redis.
i presume redis would only be needed if a load balancing SPCS shinyproxy service was created (service Max containers > 1).

i have a bunch of changes i need to test in next week to this implementation

1. fix/changed the spcs authentication to use current available roles to support access-groups[ ] and admin-groups.
2. proxies with multiple containers in a single SPCS service
3. configuring volumes on proxies
4. suspend (pause) and resume of proxies (SPCS service)

Snowpark container services further support.
add SPCS authentication single-sign-on. Snowflake account roles names can be used in shinyproxy "access-groups" or "admin-groups"

mirror additional config from here https://docs.snowflake.com/en/developer-guide/snowpark-container-services/specification-reference
add support block volumes and volume mounts to snowflake service containers.
add support stage volumes and volume mounts to snowflake service containers.
add support local volumes and volume mounts to snowflake service containers.
add support memory volumes and volume mounts to snowflake service containers.
add support for configuring secrets mapped as env variables or files.

add Pause and Resume of proxies by suspend and resume of the snowflake service. which saves compute costs.
Volumes can be used in container/image design to persist storage while paused/suspended.
deleting services with volumes attached. currently not snapshot.

some of the todo's remaining

1. merge latest containerproxy main branch
2. review pass through of SPCS ingress authentication HTTP headers to users proxy
3. regenerate snowflake openapi specs into java webclient
   HTTP client: Spring WebClient 5.1.18.
   JSON processing: Jackson 2.17.1
4. example SQL for the setup
5. example dockerfiles

Additional ideas

1. This could be used as Auto-Pause on inactivity, when shinyproxy inactivity time is disabled.
   snowflake services has this config on private service which is what a proxy container is configured as

[ AUTO_SUSPEND_SECS = <num> ]  

https://docs.snowflake.com/en/sql-reference/sql/create-service#optional-parameters

Though there is no way for the backend to tell containerproxy that the service is suspended. would need polling spcs service state. or other thoughts how that could work. can health check function set proxy status to paused?

sample shinyproxy config
now requires a warehouse (compute) config for running SQL, not everything yet can be done via REST api

• getting spcs sso user current available roles
• force delete snowflake service with volume not supported via rest api yet.

proxy:
  title: Open Analytics Shiny Proxy
  logo-url: https://www.openanalytics.eu/shinyproxy/logo.png
  landing-page: /
  heartbeat-rate: 10000
  heartbeat-timeout: 300000
  port: 8080
  authentication: spcs
  container-backend: spcs
  stop-proxies-on-shutdown: false
  recover-running-proxies: true
  admin-groups: [ <snowflake role(s) admin users are granted to> ]
  spcs:
    compute-pool: my_compute_pool
    compute-warehouse: my_warehouse

  specs:
    - id: 01_hello_sf
      access-groups: [ <snowflake roles users are member of> ]
      container-image: "/MYDB/images/shinyproxy/shinyproxy-integration-test-app:latest"
      container-cmd: [ "R", "-e", "shinyproxy::run_01_hello()" ]
      container-memory-request: 2048
      container-cpu-request: 1024
      port: 3838
      container-volumes: ["my-volume:/mnt/my-volume"]
      spcs-volumes:
        - name: my-volume
          source: block
          size: 20           # specify if memory or block is the volume source